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Abstract 


A  smart  focal  plane  array  has  been  designed  that  incorporates  the  func¬ 
tions  of  edge  extraction,  brightness  adaptation,  and  moving-object 
detection.  The  design  is  based  on  a  concept  first  presented  by  Mead  and 
Mahowald,  who  built  a  silicon  retina  that  was  modeled  on  the  distal 
portion  of  the  vertebrate  retina.  Their  chip  generated,  in  real  time,  out¬ 
puts  that  correspond  directly  to  signals  observed  in  the  corresponding 
levels  of  biological  retinas;  the  design  employed  a  resistive  network  to 
perform  signal  aggregation  between  cells. 

The  design  presented  in  this  report  is  compatible  with  complementary 
metal  oxide  semiconductor  (CMOS)  technology  and  can  be  readily  fabri¬ 
cated  through  MOSIS  (MOS  implementation  system).  The  layout  of  a 
prototype  array  was  created  with  the  MOSIS  scalable  CMOS  n-well 
analog  2.0-|xm  process.  Although  the  array  presented  here  uses  Si 
phototransistors,  the  architecture  is  applicable  to  any  semiconductor 
system  in  which  pnp  transistors  and  MOSFETs  (MOS  field-effect  transis¬ 
tors)  can  be  produced.  Therefore  it  could  be  adapted  for  use  with  GaAs 
quantum-well  infrared  photodetectors  (QWIPs)  to  produce  smart  focal 
plane  arrays  in  the  infrared.  In  addition,  the  design  is  appropriate  for 
HgCdTe  detectors  if  selected  area  growth  on  Si  is  possible.  A  preliminary 
design  in  which  each  pixel  is  connected  to  a  light-emitting  diode  (LED)  is 
also  discussed. 
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1.  Introduction 


A  conventional  image  processing  system  consists  of  a  focal  plane  array 
that  delivers  signals  corresponding  to  the  absolute  value  of  the  illumina¬ 
tion  at  each  point  in  an  image,  as  well  as  a  computer  that  extracts  features 
from  these  data.  In  this  system,  the  computational  cost  for  processing 
images  from  large  arrays  in  real  time  can  be  great.  Therefore,  it  is  desir¬ 
able  to  perform  some  "early"  processing,  before  the  data  leave  the  focal 
plane  array. 

This  sort  of  early  processing  is  also  performed  by  the  retina  in  humans 
and  other  vertebrates.  The  retina  contains  five  layers  of  cells,  in  which 
information  flows  both  vertically  (from  one  layer  to  the  next)  and  hori¬ 
zontally  (among  neighboring  cells  in  the  same  layer).  The  top  three  layers 
of  the  retina — photoreceptors,  horizontal  cells,  and  bipolar  cells — are  the 
best  understood,  and  are  the  basis  for  most  hardware  implementations  of 
the  retina. 

In  the  vertebrate  retina,  the  photoreceptor  layer  consists  of  rod  and  cone 
cells,  which  convert  the  image  light  to  electrical  signals.  The  horizontal 
cells  take  signals  from  the  photoreceptor  layer  (vertical  information  flow), 
as  well  as  from  neighboring  cells  (horizontal  information  flow).  The 
potential  of  any  horizontal  cell  is  determined  by  a  spatially  weighted 
average  of  neighboring  cells,  with  nearest  neighbors  having  the  largest 
contribution.  Each  bipolar  cell  receives  signals  from  one  photoreceptor 
cell  and  one  horizontal  cell  and  then  produces  a  signal  proportional  to  the 
difference  between  the  two.  Since  the  bipolar  cell  does  not  respond  to  the 
absolute  brightness  of  the  image,  only  to  differences  in  brightness,  the 
retina  can  take  widely  varying  amounts  of  incoming  light  and  produce  a 
signal  with  a  much  narrower  dynamic  range  that  still  captures  the  impor¬ 
tant  information  in  a  scene. ^ 

This  report  describes  a  design  for  a  smart  focal  plane  array  that  incorpo¬ 
rates  the  functions  of  edge  extraction,  brightness  adaptation,  and  moving- 
object  detection.  The  design  is  based  on  a  concept  first  presented  by  Mead 
and  Mahowald,^  who  built  a  silicon  retina  modeled  on  the  distal  portion 
of  the  vertebrate  retina.  Their  chip  generated,  in  real  time,  outputs  that 
correspond  directly  to  signals  observed  in  the  corresponding  levels  of 
biological  retinas;  the  design  employed  a  resistive  network  to  perform 
signal  aggregation  between  cells.  The  design  presented  in  this  report  is 
compatible  with  complementary  metal  oxide  semiconductor  (CMOS) 
technology  and  can  be  readily  fabricated  through  MOSIS  (MOS  imple¬ 
mentation  system). 


^M.  A.  Mahowald  and  Carver  Mead,  Milicon  retina,  in  Analog  VLSI  and  Neural  Systems,  Carver  Mead,  ed., 
Addison-Wesley  (1989). 

^C.  A.  Mead  and  M.  A.  Mahowald,  A  silicon  model  of  early  visual  processing.  Neural  Networks  1  (1988),  91-97. 
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2.  Cell  Architecture 


Most  hardware  implementations  of  the  retina  use  a  resistive  network 
between  horizontal  cells  to  perform  the  spatial  averaging.  However, 
resistors  are  difficult  to  implement  in  very-large-scale  integration  (VLSI) 
technology.  The  design  reported  here  relies  on  carrier  diffusion  between 
the  horizontal  cells  to  perform  the  spatial  averaging  function.^  Carrier 
diffusion  is  usually  undesirable  in  conventional  charge-coupled  device 
(CCD)  arrays,  because  it  leads  to  crosstalk  between  the  pixels.  However, 
in  this  application,  we  desire  crosstalk  between  the  horizontal  cells.  The 
horizontal  cells  are  therefore  designed  to  have  a  large  amount  of  crosstalk 
by  being  placed  all  in  one  big  n-well  with  a  common  base. 

3.  Cell  Layout 

Figures  1  and  2  show  one  pixel  of  the  smart  focal  plane  array.  The  layout 
was  accomplished  with  L-EDIT^^,  a  powerful  PC-based  program  for 
VLSI  design.  Each  pixel  contains  one  photoreceptor  cell  and  one  horizon¬ 
tal  cell.  The  cells  are  pnp  phototransistors,  with  the  base  current  provided 
by  the  photocurrent  produced  in  the  base-collector  region.  The  MOS 
transistors  are  used  to  select  the  pixel  for  readout.  The  photoreceptor  cell 
is  isolated  from  neighboring  cells  by  being  surrounded  with  a  reverse- 
bias  pn  junction.  The  emitters  are  held  at  virtual  ground.  The  horizontal 
cell  is  placed  in  an  n-well  that  covers  most  of  the  pixel,  and  is  shared  by 
the  other  horizontal  cells  in  the  array.  A  metal-defined  window  is  placed 
over  the  base  regions  of  the  pnp  phototransistors  to  allow  roughly  equal 
illumination  of  the  photoreceptor  and  horizontal  cells. 

The  photons  incident  on  the  n-well  base  region  create  electron-hole  pairs, 
and  any  holes  that  diffuse  towards  the  reverse-biased  collector-base 
region  are  swept  towards  the  collector  by  the  electric  field.  The  rest  of 
these  holes  combine  with  excess  electrons.  The  photons  incident  on  the 
base-collector  depletion  region  also  create  electron-hole  pairs,  which  are 
separated  by  the  electric  field  in  this  region,  with  holes  going  to  the 
collector  and  electrons  towards  the  base.  The  result  is  that  there  are  excess 
electrons  in  the  n-well  base  of  the  photoreceptor  and  horizontal  cells.  In 
the  photoreceptor  cell,  this  base  current  creates  an  emitter  current 
In  the  horizontal  cell,  some  of  these  electrons  will  diffuse  out  in  the  large 
n-well  base  towards  other  pixels,  creating  emitter  current  in  neighboring 
pixels  as  well  as  in  the  local  pixel.  Some  of  these  electrons  will  recombine 
before  they  reach  another  emitter.  This  effect  can  be  modeled  as  a  base 
resistance. 

Consider  the  situation  of  only  one  pixel  illuminated.  The  effective  resis¬ 
tance  from  the  local  horizontal  cell  to  neighboring  cells  is  proportional  to 
the  distance  between  them.  The  collectors  (p-substrate)  of  all  the  cells  are 
held  at  the  same  negative  potential.  The  emitters  are  held  at  virtual 

^C.  Wu  and  C.  Chiu,  A  new  structure  of  the  2-D  silicon  retina,  IEEE  J.  Solid  State  Circuits  30,  No.  8  (August 

1995). 
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ground.  Therefore  the  base  emitter  potential  of  any  neighboring  cell  is 
of  the  local  cell  minus  the  potential  drop  across  the  base  resistance. 
This  potential  drop  increases  with  increasing  distance  from  the  local 
illuminated  cell.  Since  the  emitter  current  follows  the  Ebers-Moll 
equation, 

le  =  Isat  exp{Vbe/kT), 

the  ratio  of  emitter  current  in  a  neighboring  cell  that  comes  from  the 
illuminated  cell  would  be  exp{-qR^L/kT),  where  is  the  effective  base 
resistance  per  unit  length  and  L  is  the  distance  between  the  illuminated 
cell  and  the  neighboring  cell.  This  is  the  spatial  averaging  that  is  desired 
for  the  horizontal  cells. 


Figure  1.  Layout  of 
one  pixel. 


Figure  2.  Cross- 
sectional  view  of 
pixel. 


p-substrate 
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4.  Array  Layout 

In  the  hardware  implementation  of  the  retina  presented  here,  the  third 
layer  (the  bipolar  cells)  is  implemented  by  subtraction  of  the  emitter 
currents  in  the  photoreceptor  cell  and  the  horizontal  cell  of  each  pixel.  A 
prototype  11  x  11  array  is  shown  in  figure  3.  The  pixels  are  individually 
addressable,  according  to  the  same  scheme  as  in  Wu  and  Chiu.^  The 
circuit  diagram  for  this  prototype  is  shown  in  figure  4.  When  the  system 
selects  a  row  by  taking  the  corresponding  control  line  high,  the  emitter 
currents  of  all  the  pixels  in  that  row  are  connected  to  lines  that  go  to  the 
output  through  a  column  switch.  There  are  separate  lines  for  photorecep¬ 
tor  cell  current  and  horizontal  cell  current.  If  the  row  is  not  selected,  the 
emitter  currents  are  shunted  to  ground.  If  the  column  switch  is  on,  the 
emitter  current  is  connected  to  the  output;  otherwise,  it  is  shunted  to 
ground.  The  inputs  to  the  array  are,  therefore,  11  row  control  lines,  11 
column  control  lines,  positive  supply,  negative  supply,  and  ground.  The 
outputs  are  two  currents,  corresponding  to  the  emitter  current  in  the 
photoreceptor  cell  and  the  horizontal  cell  for  the  pixel  selected. 


Figure  3. 11  x  11  array, 
including  switching 
transistors. 

4- 


Figure  4.  Circuit 
diagram  for 
addressing  pixel  by 
row  and  column. 


^C.  Wu  and  C.  Chiu,  A  new  structure  of  the  2-D  silicon  retina,  IEEE  J.  Solid  State  Circuits  30,  No.  8  (August 
1995). 
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5.  Amplifier  Design 

The  current  from  the  cells  is  not  sufficient  to  drive  output  lines.  Also,  the 
design  requires  that  the  emitters  be  held  near  ground  (virtual  ground). 
Thus,  the  design  incorporates  a  current  to  voltage  amplifier  (fig.  5)  that 
keeps  the  emitters  near  ground,  taken  from  Wu  and  Chiu.^  The  gain  can 
be  controlled  by  VI  and  V5.  Figure  6  shows  the  amplifier's  physical 
layout,  and  figure  7  shows  a  SPICE  simulation  of  its  characteristics.  The 
prototype  device  contains  two  of  these  amplifiers,  one  for  photoreceptor 
(vertical)  cell  current  and  one  for  horizontal  cell  current. 

Edge  detection  is  provided  by  the  retina  architecture.  To  see  this,  consider 
an  image  with  a  large  edge  contrast.  Eigure  8  shows  the  output  current 
for  the  photoreceptor  cells  and  the  horizontal  cells  in  this  case.  Since  the 
photoreceptor  cells  are  isolated  from  each  other,  the  output  of  these  cells 
also  exhibits  a  large  contrast.  The  horizontal  cell  output  is  a  smoothed 
version  of  the  photoreceptor  output,  where  the  smoothing  is  due  to  the 
spatial  averaging  in  the  horizontal  cells.  The  difference  signal,  therefore, 
is  maximized  near  an  edge. 


Figure  6.  Layout  of 
current  to  voltage 
amplifier. 
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^C.  Wu  and  C.  Chiu,  A  new  structure  of  the  2-D  silicon  retina,  IEEE  J.  Solid  State  Circuits  30,  No.  8  (August 
1995). 
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Figure  7.  Result  of 
SPICE  simulation  for 
amplifier. 


Figure  8.  Theoretical 
output  in  one 
dimension  for  an 
image  with  a  sharp 
edge. 


2  6  10  14  18 

Circuit  in  (A) 


Pixel  position 

The  retina  architecture  also  provides  an  enhanced  signal  for  moving 
objects.  This  enhancement  is  due  to  the  difference  in  response  time  of  the 
photoreceptor  and  horizontal  cell  networks.  It  takes  some  time  for  the 
electrons  to  diffuse  out  to  neighboring  pixels  in  the  horizontal  cell  net¬ 
work.  This  time  is  called  the  settling  or  integration  time.  The  integration 
time  for  the  photoreceptor  cells  is  the  integration  time  required  for  a 
single  pnp  phototransistor,  which  is  shorter  than  for  the  horizontal  net¬ 
work,  because  these  photoreceptor  cells  are  isolated  from  each  other. 
Therefore,  when  the  image  presented  to  the  focal  plane  changes,  the 
difference  signal  between  the  two  networks  will  initially  be  large.  It  will 
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Figure  9.  Circuit 
diagram  for  an  array 
with  output  from 
each  pixel  driving  an 
LED. 


Figure  10.  Conceptual 
diagram  for  an  array 
with  pixel  output 
driving  an  LED. 


then  decrease  after  the  settling  time  of  the  horizontal  network.  By  detect¬ 
ing  these  large  current  pulses,  the  system  can  detect  a  moving  object. 

A  variation  of  this  architecture  can  also  be  envisioned  where  each  pixel  in 
the  array  is  connected  to  a  light-emitting  diode  (LED).  In  this  case,  the 
transistors  used  to  select  the  pixels  would  be  replaced  by  a  pair  of 
MOSFETs  and  a  bipolar  transistor  that  would  amplify  the  emitter  current 
difference  and  drive  the  LED.  If  the  LED  operated  in  the  visible,  the  array 
output  would  act  like  a  display.  Figure  9  is  a  circuit  diagram  for  this 
architecture,  and  figure  10  is  a  conceptual  diagram,  where  the 
beamsplitter  is  a  narrow-band  reflector  that  has  high  reflectivity  at  the 
LED  output  wavelength. 


Kc 


focal  plane,  conceptual 


□  detector  +  electronics 
O  LED 
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6.  Conclusion 


A  silicon  retina  design  is  presented  that  can  perform  the  functions  of  edge 
extraction,  brightness  adaptation,  and  moving-object  detection.  This 
design  is  an  improvement  over  earlier  designs,  which  employed  a  resis¬ 
tive  network  to  connect  the  cells.  Because  the  resistive  network  is  elimi¬ 
nated,  the  chip  design  is  more  easily  realized  with  a  CMOS  process. 
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